Looking for patterns in missing data

row / column patterns

• Do all variables have the same percentage of values missing?

• Are there row patterns to the missing values?

Create a dataset with missing values

Row / column missing patterns

Missing values by column

colSums(is.na(mycars)) %>%
  sort(decreasing = TRUE)

## gear disp   hp drat   am  cyl qsec   vs  mpg   wt carb 
##   32   12   12   11    3    1    1    1    0    0    0

Row / column missing patterns

Missing values by row

rowSums(is.na(mycars)) %>%
  sort(decreasing = TRUE)

##          Merc 450SE         Merc 450SLC Lincoln Continental            Fiat 128 
##                   5                   5                   5                   5 
##            Merc 280           Merc 280C          Merc 450SL  Cadillac Fleetwood 
##                   4                   4                   4                   4 
##   Chrysler Imperial         Honda Civic      Toyota Corolla        Lotus Europa 
##                   4                   4                   4                   3 
##         AMC Javelin           Fiat X1-9           Mazda RX4       Mazda RX4 Wag 
##                   2                   2                   1                   1 
##          Datsun 710      Hornet 4 Drive   Hornet Sportabout             Valiant 
##                   1                   1                   1                   1 
##          Duster 360           Merc 240D            Merc 230       Toyota Corona 
##                   1                   1                   1                   1 
##    Dodge Challenger          Camaro Z28    Pontiac Firebird       Porsche 914-2 
##                   1                   1                   1                   1 
##      Ford Pantera L        Ferrari Dino       Maserati Bora          Volvo 142E 
##                   1                   1                   1                   1

Row / column missing patterns

heatmap

geom_tile()

Row / column missing patterns

heatmap

mi::missing_data.frame()

library(mi)
x <- missing_data.frame(mycars)

image(x)

(gear not shown since all are missing)

Row / column missing patterns

Missing values by variable

geom_tile()

Row / column missing patterns

Missing values by variable

geom_tile() with standardized variables

Row / column missing patterns

Missing values by variable

reordered by number of missing

Row / column missing patterns

Missing values by variable

Row / column missing patterns

Missing values by variable

missing patterns

x <- mi::missing_data.frame(mycars)

## NOTE: In the following pairs of variables, the missingness pattern of the second is a subset of the first.
##  Please verify whether they are in fact logically distinct variables.
##      [,1]   [,2]  
## [1,] "disp" "drat"
## [2,] "disp" "qsec"
## [3,] "hp"   "drat"
## [4,] "hp"   "qsec"
## [5,] "hp"   "am"  
## [6,] "drat" "qsec"

class(x)

## [1] "missing_data.frame"
## attr(,"package")
## [1] "mi"

x@patterns

##  [1] nothing              nothing              nothing             
##  [4] nothing              nothing              nothing             
##  [7] nothing              nothing              nothing             
## [10] disp, hp, drat       disp, hp, drat       disp, hp, drat, am  
## [13] disp, hp, drat       disp, hp, drat, qsec disp, hp, drat      
## [16] disp, hp, drat, am   disp, hp, drat       cyl, disp, hp, drat 
## [19] disp, hp, drat       disp, hp, drat       nothing             
## [22] nothing              vs                   nothing             
## [25] nothing              disp                 nothing             
## [28] hp, am               nothing              nothing             
## [31] nothing              nothing             
## 8 Levels: nothing vs disp hp, am disp, hp, drat ... cyl, disp, hp, drat

levels(x@patterns)

## [1] "nothing"              "vs"                   "disp"                
## [4] "hp, am"               "disp, hp, drat"       "disp, hp, drat, am"  
## [7] "disp, hp, drat, qsec" "cyl, disp, hp, drat"

summary(x@patterns)

##              nothing                   vs                 disp 
##                   18                    1                    1 
##               hp, am       disp, hp, drat   disp, hp, drat, am 
##                    1                    7                    2 
## disp, hp, drat, qsec  cyl, disp, hp, drat 
##                    1                    1

Row / column missing patterns

Missing values by variable

missing patterns

(repeated patterns are reduced to one row)

visna() w/ row sort

Sorted by most common to least common missing pattern (top to bottom)

visna() w/ column sort

Sorted by variable with the most to least missing values (left to right)

visna() w/ row & column sort

NYC School data

## [1] 460   6

NYC School data

Looking for patterns in missing data

value patterns

• Are missing patterns correlated with values of another variable?

• Are certain value ranges more likely to be missing?

• Or are there no patterns at all?

NYC School data

Does the proportion of schools with missing data vary by borough?

Data: SAT2010.csv

Missing by borough

Missing by borough

K, Q, M, X, R

Manhattan

Brooklyn

Queens

The Bronx

Staten Island

Missing by borough

K, Q, M, X, R

Manhattan = New York County

Brooklyn = Kings County

Queens = Queens County

The Bronx = Bronx County

Staten Island = Richmond County

Missing by borough

K, Q, M, X, R

Manhattan = New York County = “M”

Brooklyn = Kings County = “K”

Queens = Queens County = “Q”

The Bronx = Bronx County = “X”

Staten Island = Richmond County = “R”

Missing by borough

Missing by borough

Missing by borough and scores

EXAMPLE: Snowfall

I need to know how much snow fell per day in New York State in February 2017, on a county or more detailed level. I know some government agency measures and reports snowfall and puts the data online.

Snowfall

I need to know how much snow fell per day in New York State in February 2017, on a county or more detailed level. I know some government agency measures and reports snowfall and puts the data online.

Source: https://www.ncdc.noaa.gov/snow-and-ice/daily-snow/NY/snowfall/20170201

Accessed: 2017-10-26

Number missing by county mean

Number missing by day of month

Number of missing values per day by total snowfall

no gridlines